Interfacing with a computing application using a multi-digit sensor

ABSTRACT

A technology is described for interfacing with a computing application using a multi-digit sensor. A method may include obtaining an initial stroke using a single digit of a user on the multi-digit sensor. A direction change point for the initial stroke can be identified. At the direction change point for the initial stroke, a number of additional digits can be presented by the user to the multi-digit sensor. Then a completion stroke can be identified as being made with the number of additional digits. A user interface signal to can be sent to the computing application based on the number of additional digits used in the completion touch stroke. In another configuration of the technology, the touch stroke or gesture may include a single stroke where user interface items can be selected when additional digits are presented at the end of a gesture.

BACKGROUND

Selecting items from a menu of functions is a basic computer interactiontask when a user interface is used by an end user. Most current menudesigns assume that the user has a single interactive point for inputwhich means the menu can be traversed in a linear fashion. For example,a user may have a mouse cursor or a stylus which allows the user toselect menu items one by one. In general, such menus can be activated byclicking on an item or by dragging and/or crossing the interaction pointthrough the menu item. Dragging or crossing the interaction pointthrough the menu item is often used with a stylus for a marking menu ora crossing menu system.

In a system with many menu options, the menus are often nested and thismeans opening a menu item may open a new sub-menu. The use of nestingmenus forces the user to traverse many menu items before arriving attheir final choice. This process is comparatively slow and potentiallyerror-prone.

In recent years, computer interface hardware has been produced that canrecognize when a user presents the user's digits to a sensor device. Forexample, certain hardware can recognize when a user touches multiplepoints on a computer screen, interface pad, or interface surface. Thesemulti-digit recognition systems that can register multiple digits (e.g.,multiple touch points) for the user interface can be called multi-digitinterface systems. In such multi-digit interface systems, the menudesign has not taken advantage of the multiple-points of interaction toenhance the menu interaction. As a result, the menus used in multi-digitinterface systems have generally remained the same as in the singledigit or single interaction point interface systems.

SUMMARY

This summary is provided to introduce a selection of concepts in asimplified form that are further described below in the detaileddescription. This summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter. While certaindisadvantages of prior technologies are noted above, the claimed subjectmatter is not to be limited to implementations that solve any or all ofthe noted disadvantages of the prior technologies.

Various embodiments are described for interfacing with a computingapplication using a multi-digit sensor. One method may include obtainingan initial stroke using a single digit of a user on the multi-digitsensor. A direction change point for the initial stroke can beidentified. At the direction change point for the initial stroke, anumber of additional digits can be presented by the user to themulti-digit sensor. Then a completion stroke can be identified as beingmade with the number of additional digits. A user interface signal tocan be sent to the computing application based on the number ofadditional digits used in the completion stroke. In anotherconfiguration of the technology, the stroke or gesture may include asingle stroke where user interface items can be selected when additionaldigits are presented at the end of a gesture.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a system for interfacing with acomputing application using a multi-digit sensor.

FIGS. 2A-2E illustrate an embodiment of a group of gestures that can beused for interfacing with a computing application using a multi-digitsensor.

FIG. 3 is a flowchart illustrating an embodiment of a method ofinterfacing with a computing application using a multi-digit sensor.

DETAILED DESCRIPTION

Reference will now be made to the exemplary embodiments illustrated inthe drawings, and specific language will be used herein to describe thesame. It will nevertheless be understood that no limitation of the scopeof the technology is thereby intended. Alterations and furthermodifications of the features illustrated herein, and additionalapplications of the embodiments as illustrated herein, which would occurto one skilled in the relevant art and having possession of thisdisclosure, are to be considered within the scope of the description.

The present technology provides the use of a set of user digit gesturesor digit strokes that can reduce the amount of existing menu nesting byenabling a user to more directly select from a larger set of menuoptions. The gestures can be composed of a finger or hand stroke incombination with a varying number of user digits sensed by a multi-digitsensor. Examples of a multi-digit sensor can include a multi-touchsurface device, multi-touch screen, or an optical sensor. The digitsused by the user can include the fingers and thumbs. The number of digitcontacts together with the shape of the gesture determines which of theseveral options may be invoked. These gestures are straight-forward toremember, perform, and be detected by the system. In addition, thegestures are also distinct enough not to be frequently randomly invokedor confused with other gestures used for interfacing with a computingdevice.

FIG. 1 illustrates a system for interfacing with a computing applicationusing a multi-digit sensor 100. The system may include a multi-digitsensor capable of sensing multiple digits (e.g. fingers and thumbs) of auser. In one example configuration, a multi-touch surface device 108 canbe configured to detect multiple input touch points from a user 114 whenthe user touches the user's digits to the sensing surface. Themulti-touch surface device may be a multi-touch pad attached as aperipheral or an integrated multi-touch screen device. Many multi-touchdevices use capacitive sensing technology, while other multi-touchdevices may use optical reflection or other existing touch sensingtechnologies.

Another example type of a multi-digit sensor may be a motion and/ordepth sensor 120 that can be used to sense a user's gestures 124. Anexample depth sensor may include an infrared projector that projects aninfrared beam 122 combined with an optical sensor (e.g. CMOScomplementary metal-oxide-semiconductor chip). The sensing range of thedepth sensor may be adjustable, so the sensor can automatically depthcalibrate in response to the user's physical environment, such as thepresence of furniture. Infrared may also allow the multi-digit sensor tosense a user's digits under many types of ambient light conditions.Other types of optical sensors may be used such as a laser sensors,light sensors, or visual cameras.

A computing application 102 can be configured to receive input through auser interface or a graphical user interface 104. The computer softwareapplication can be executing on one or more processors 112. For example,the computer software application can be executing on a desktopworkstation, server, handheld computing device or another type ofcomputing device. The graphical user interface that is stored in acomputer memory 110 may be displayed to an end user via a display output119 or display screen. In one embodiment, the display output may be anindependent output device or the display output may have the multi-touchsurface attached directly over the display output. The user interfacemay also be an audio interface, a touch interface or another interfacethat is perceptible to a human and where menu sub-items can be chosen.

A multi-digit recognition module 106 can be in communication with thegraphical user interface 104. The multi-digit recognition module can beconfigured to obtain and interpret the electronic signals from themulti-digit sensor device. In one embodiment, the multi-digitrecognition module can be a device driver in communication with thegraphical user interface and computing application through an operatingsystem.

The input signals received by the multi-digit recognition module caninclude signals representing a gesture and a number of additional digitspresented by the user on the multi-digit sensor device. The term“presented” as used here is defined as a user providing their digits tothe digit sensor so that the sensor can recognize a gesture, hand sign,or number of fingers in a gesture. The additional digits may be theadditional three fingers and/or thumb on the user's hand. For example,the gesture may be a user moving one finger in a straight or curved lineaway from an initial point or starting point. In addition, the gesturemay be an L-shape or another shape with corners that indicates that theuser desires to provide a command to the computing application.

A user interface item 116 in the user interface can be configured toactivate a selected user interface function in response to the gestureand the number of digits communicated to the user interface by themulti-digit recognition module. The multi-digit capabilities of themulti-digit sensor device can be combined together with the gesturingmenu design and additional digits for selecting user interface items toextend the number of directly accessible menu items with a single handstroke and without menu nesting.

This technology provides the ability to cull the menu options displayedbased on the combination of the direction of the movement of a gestureand the number of digits presented to the multi-digit sensor device. Forexample, first move one finger to the left, followed by two fingers downto select a particular sub-option of the left menu. In the case of amulti-touch surface this gesture can be performed without lifting thehand from the surface which increases the speed of the selection. Timespent searching through multiple levels of menus can also be reducedbecause more menu items are available at a time which flattens theoverall menu structure.

FIGS. 2A-2E illustrates an example of gestures that can be used with thepresent technology. In this example, the gesture can select a userinterface function and the presentation or placement of the additionaldigits can enable selection of sub-options or items in the userinterface function. The depicted menu selection process can extend amarking menu with the use of multi-digit gestures. The first hand with asingle finger extended in each figure shows the starting position foreach hand movement. Then the second hand position shows a number ofdigits or fingers that may be added to the gesture at a direction changepoint 240 in FIG. 2A. Both hands represent a single user's hand (in thiscase their right hand).

As illustrated in FIG. 2A, the user's finger can be moved away from theinitial point 202 and the first straight horizontal line 204 may selectthe type of function that will be undertaken. As an example, the firstline of the gesture that is traced in a direction to the user's rightmay represent that an editing function is desired.

When the user stops at the end of the line, a direction change may beexpected by the interface system. FIG. 2B illustrates that one or moreadditional menu items 210, 212 or sub-options can be displayed inresponse to the user's finger tips to help prompt the user regardingwhich items are available. When a multi-touch surface device is beingused, the sub-options can be displayed under the user's finger tips. Inthe case of an edit function, items such as copy, copy format, cut,paste, or paste special may be the additional items or sub-options. Thistype of prompting may be provided for users who are new to the interfacesystem, while experts at using this interface may not need graphicalprompts. In this example, up to five user digits can be sensed by themulti-digit recognition module. These five different fingerconfigurations, illustrated in FIGS. 2A-2E, can be used to determinewhich of five user interface items are selected for the gesture. Thedirection the gesture is made away from an initial point determineswhich set of five user interface items will be used to make a furtherselection. Each different direction can have a different item group orset to present to the user. The additional selection can be based on thenumber of digits sensed by the multi-digit recognition module.

The L-shaped gestures in this example gesture are comparativelyuncomplicated to perform, learn, and recognize. These gestures are alsoeasily differentiated from other that gestures may be used to interactwith the computing device. The L-shaped gestures have been previouslyused with a single contact stylus and in the past offered only a singleinvocation per hand stroke. Whereas, the present technology can offer atleast two invocation options per stroke.

As illustrated by FIGS. 2A-2E, the gesture may form a backwards L shape.The initial stroke can be created by the user by moving the single digitaway from an initial point. After the initial stroke is completed, achange direction point can be reached. A change direction point is wherethe direction of a gesture or stroke changes direction at an inflectionpoint or corner. Next, a completion stroke 220 (in FIG. 2C or FIG. 2A)can be performed by the user by moving perpendicular to the initialstroke to form a backwards L. Alternatively, the user may also form aforward L, sideways L facing down, or a sideways L facing up dependingon the sub-menu desired to be selected. While the L shape is illustratedhere as being orthogonal to a reference side of the sensor device, anyrotated orientation of L shape can be used and recognized by the sensor.

While using an L shape is beneficial because it is unlikely to beconfused with other interface gestures, the completion stroke can beperformed by moving away at another angle to the initial stroke. Insteadof moving in a different direction at a 90 degree angle, the completionstroke can move away from the change direction point at some other anglebetween 0 and 90 degrees. An example of this can be performing thecompletion stroke at a 45 degree angle to the left or right of theinitial stroke.

The L-shaped gestures have been used previously with an interface knownas Hover Widgets which were developed by Microsoft. However, the presenttechnology adds the ability to invoke one of a plurality of possibleoptions depending on how many fingers the user has presented to themulti-digit sensor at the direction change point. In the example case,this change direction point is at the corner 240 of the L gesture as inFIG. 2A.

While L-shaped strokes in different directions have been described, thegesture can be other shapes too. For example, defined curved or loopingshapes can be used to pick the function and then multiple user digitscan be presented to the multi-digit sensor device to select items orsub-options. For example, a partial spiral shape with an inflection onthe end may be used. Alternatively, a shape that has multiple changedirection points or corners can be used such as a forward or reverse Zshape or a V type shape. Further, the initial stroke may be anotherdistinctive shape that is differentiated from other user interface inputstrokes.

In some embodiments, the initial stroke can be performed by an East orWest stroke with respect to the surface plane of multi-digit sensordevice. Then the completion stroke can be performed by moving in a Northor South direction with respect to the multi-digit sensor device.Alternatively, the initial stroke can be performed in the North or Southdirection and the completion stroke can be performed in the East or Westdirection.

The present technology can use a sequence of one or more hand strokesand a particular number of digits moving in synchrony relative to themulti-digit sensor to invoke a menu option or menu item. When theadditional number of digits is presented by the user, the extra digitscan move along with part of the hand stroke or gesture from the changedirection point to send a user interface signal to the computerapplication or to complete the selection of the user interface item. Asdiscussed, the technology described can extend the possible space ofuser interface options that are directly selectable in one user actionby a factor of at least five and in some cases 10 or even 20 usingdirectionality. This increase in user selection options at each levelcan reduce the amount of finger movement on the screen in cases of verylarge (e.g., deep) menus.

This technology may use a single finger to begin the gesture, butpossibly multiple fingers to complete the selection. The expandedselection options allows the system to cull down the number of optionsdisplayed on a user interface screen or under a user's fingers on amulti-touch screen, thus reducing visual clutter and providing a simplerand faster way to find and select an item. For example, as soon as theuser moves to the user's right (East), the other three options (North,South, and West) can fade away and the sub-options can appear. Ifperformed sufficiently fast in expert use, the user interfacevisualization of the menu can completely disappear. However, the usercan still be sure that a correct menu was selected by knowing how manydigits or fingers the user presented to the multi-digit sensor.

While FIGS. 2A-2E illustrate some examples of possible paths for thegestures and presented digits, similar drawings can be generated for allpaths that can be traveled in an upward, downward, left and right(North, South, East and West) directions. Additional digits can bepresented on such paths to directly select additional sub-options oritems. This extends the available menu options at least five times inthe case where a single hand is considered or more if an additional handis used.

FIG. 3 is a flowchart illustrating a method of interfacing with acomputing application using a multi-digit sensor device. The method caninclude the operation of obtaining an initial stroke using a singledigit of a user on the multi-digit sensor device, as in block 310. Thesingle digit stroke may be reserved for initiating a command andselecting a starting function point in the user interface menu. Adirection change point or intermediate stopping point for the initialstroke can be identified, as in block 320. At this direction changepoint or inflection point, the system may be configured to expectadditional input. For example, a number of additional digits (e.g.,fingers and/or thumbs) may be placed by the user on the multi-touchsurface device, as in block 330. These additional digits can bepresented when the direction change point at the end of the initialstroke with the single digit is reached. The system may identify anadditional two digits (or more) that have been presented to themulti-digit sensor device.

A completion stroke with the number of additional digits can also beidentified as in block 340. A user interface signal to can be sent tothe computing application or a user interface item can be selected inthe computing application based on the number of additional digits usedin the completion stroke, as in block 350.

In one embodiment, the number of fingers and thumbs can be used todefine which menu item is selected. In an alternative configuration, thenumber of additional menu sub-options displayed at the end of thecompletion stroke can be reduced based on the number of additionaldigits used by the user. In the case of reduced menu items at the end ofthe stroke, an additional selection may be made to determine the finalsub-menu or sub-item selection.

In another configuration of the technology, the gesture stroke can beobtained based on a single digit placed on the multi-digit sensor deviceby a user. When the stopping point for the single gesture stroke on themulti-digit sensor device is identified, then the menu sub-items can beselected when the gesture has stopped. Rather than obtaining a secondpart of the gesture stroke, the number of additional digits presented bythe user to the multi-digit sensor device at the end of the gesturestroke can be determined. As a result, a user interface signal to can besent to a computing application or a user interface item in thecomputing application can be selected based on the number of additionaldigits presented by the user at the stopping point. In contrast to theprevious embodiment which has a direction change point, this selectioncan be made with just one gesture. However, simpler gestures can createmore false positives when identifying the gesture. For example, after asingle line or curve stroke has stopped, then one or more digits can beplaced on the multi-digit sensor device. The additional digits canselect an item from the user interface. The single line stroke may alsobe straight or curved.

Described in other terms, the single gesture of this example selects adefined user interface function and then the placement of the additionaldigits when the gesture is complete enables selection of sub-items forthe user interface function. As discussed before, at least five userdigits can be sensed by the multi-digit recognition module and used todetermine which of five user interface sub-items is selected for thatgesture. The direction of the gesture may be made away from an initialpoint, and the gesture direction determines which set of five userinterface sub-items will be supplied to make a further selection basedon the number of digits sensed by the multi-digit sensor device.

In an additional configuration of the technology, both hands can be usedto select menu items. When the single gesture is made by either hand,all the remaining nine digits of both hands can be used to select asub-item when the gesture is completed. Using nine digits can increasethe capacity of the menu system by a factor of at least nine andsometimes more using stroke directionality.

In another example interface, cascading of the multi-stroke menus can beprovided. A user may first select a sub-menu using the L stroketechnique described, and then the user may proceed to select optionsfrom that submenu by presenting additional fingers. In a similar manner,two or more L strokes can be chained together to open multiple sub-menusand then the final sub-menu selection can be made when the user presentsadditional sub fingers to select an item in the last sub-menu that wasopened.

To reiterate, the user can begin a menu initiation gesture with a singlefinger presented to a multi-digit sensor. When the user reaches thecorner point or direction change point of the L gesture, the user canpresent additional digits to specify two to nine or more sub-options oritems and directly select a menu item by moving all the placed digits(e.g., fingers and thumbs) in relation to the surface to complete thegesture.

Some of the functional units described in this specification have beenlabeled as modules, in order to more particularly emphasize theirimplementation independence. For example, a module may be implemented asa hardware circuit comprising custom VLSI circuits or gate arrays,off-the-shelf semiconductors such as logic chips, transistors, or otherdiscrete components. A module may also be implemented in programmablehardware devices such as field programmable gate arrays, programmablearray logic, programmable logic devices or the like.

Modules may also be implemented in software for execution by varioustypes of processors. An identified module of executable code may, forinstance, comprise one or more blocks of computer instructions, whichmay be organized as an object, procedure, or function. Nevertheless, theexecutables of an identified module need not be physically locatedtogether, but may comprise disparate instructions stored in differentlocations which comprise the module and achieve the stated purpose forthe module when joined logically together.

Indeed, a module of executable code may be a single instruction, or manyinstructions, and may even be distributed over several different codesegments, among different programs, and across several memory devices.Similarly, operational data may be identified and illustrated hereinwithin modules, and may be embodied in any suitable form and organizedwithin any suitable type of data structure. The operational data may becollected as a single data set, or may be distributed over differentlocations including over different storage devices. The modules may bepassive or active, including agents operable to perform desiredfunctions.

Furthermore, the described features, structures, or characteristics maybe combined in any suitable manner in one or more embodiments. In thepreceding description, numerous specific details were provided, such asexamples of various configurations to provide a thorough understandingof embodiments of the described technology. One skilled in the relevantart will recognize, however, that the technology can be practicedwithout one or more of the specific details, or with other methods,components, devices, etc. In other instances, well-known structures oroperations are not shown or described in detail to avoid obscuringaspects of the technology.

Although the subject matter has been described in language specific tostructural features and/or operations, it is to be understood that thesubject matter defined in the appended claims is not necessarily limitedto the specific features and operations described above. Rather, thespecific features and acts described above are disclosed as exampleforms of implementing the claims. Numerous modifications and alternativearrangements can be devised without departing from the spirit and scopeof the described technology.

The invention claimed is:
 1. A method performed by a computing devicecomprising a display and an input device, the method comprising:monitoring discrete unbroken strokes inputted by a user of the computingdevice to detect, among the strokes, first unbroken strokes and secondunbroken strokes, each discrete unbroken stroke comprising anuninterrupted input inputted by the user providing input from only asingle finger; the monitoring detecting the first strokes by analyzingfeatures of the strokes to determine that the first strokes havefeatures that correspond to the first strokes: (i) having been inputtedby only a single finger, and (ii) having ended with the single fingerhaving moved substantially in only two predefined directions, and (iii)having never been accompanied by input from any other fingers, the twopredefined directions comprising a first direction and then a seconddirection, the first direction different than the second direction; eachtime a first stroke is detected, responding by automatically activatinga first function on the computing device; the monitoring detecting thesecond strokes by analyzing features of the strokes to determine thatthe second strokes have features that correspond to (i) having beeninputted by only a single finger, and (ii) having ended with the singlefinger having moved substantially in only the two directions, and (iii)having been accompanied by input from only one other finger that movedsubstantially in only the second direction; and each time a secondstroke is detected, responding by automatically activating a secondfunction on the computing device.
 2. A method according to claim 1,further comprising: the monitoring detecting third strokes by analyzingfeatures of the strokes to determine that the third strokes (i) wereinputted by only a single finger, and (ii) ended with the single fingerhaving moved substantially in only the two directions, and (iii) wereaccompanied by input from only two other fingers that movedsubstantially in only the second direction; and each time a third strokeis identified, responding by automatically activating a third functionon the computing device.
 3. A method according to claim 1, wherein thestrokes are inputted with a touch-sensitive surface.
 4. A methodaccording to claim 1, wherein the strokes are inputted with a videocamera.
 5. A method according to claim 1, wherein the first functioncomprises displaying a menu and the second function comprises an item ofthe menu.
 6. A method according to claim 1, wherein the first directionis substantially perpendicular to the second direction.
 7. A methodaccording to claim 1, wherein the computing device comprises a displaywith two pairs of opposing edges, and the first direction corresponds tothe first pair of opposing edges and the second direction corresponds tothe second pair of opposing edges.
 8. A method performed by a computingdevice comprising a display and an input device, the method comprising:receiving strokes inputted by a user using the input device, the inputdevice detecting presentation of fingers for input and detectingmovement of fingers presented for input; analyzing the strokes inputtedby a user with the input device to identify single unbroken strokessubstantially having an “L” shape, such single unbroken “L” shapedstrokes having a first segment and a substantially perpendicular secondsegment, wherein for each of the strokes only a single finger inputs thefirst and second segments, for first of the strokes no other fingeraccompanies the single finger when inputting the corresponding secondsegments, and for second of the strokes another finger accompanies thesingle finger when inputting the corresponding second segments; for theunbroken “L” shaped strokes, determining whether another fingeraccompanied the single finger when inputting the second segment; andeach time an unbroken “L” shaped stroke ends, automatically selecting,from among a first function and a second function according to whetheranother finger was determined to accompany the single finger.
 9. Amethod according to claim 8, wherein the first function causes a firstset of menu items to be displayed and the second function causes asecond set of menu items to be displayed.
 10. A method according toclaim 8, wherein the first function activates a first user interfacefeature and the second function activates a second user interfacefunction.
 11. A method according to claim 8, further comprising:selecting the first function each time a unbroken “L” shaped stroke endswith a count of zero, selecting the second function each time a unbroken“L” shaped stroke ends with a count of one, and selecting a thirdfunction each time a unbroken “L” shaped stroke ends with a count oftwo.
 12. A method according to claim 8, wherein the “L” shaped strokescan have any arbitrary orientation relative to the display.
 13. A methodaccording to claim 8, further comprising displaying a menu when an “L”shaped stroke ends and which items are displayed in the menu depends onthe count of the “L” shaped stroke.
 14. A method performed by acomputing device comprising a display and an input device, the methodcomprising: analyzing features of discrete unbroken strokes, inputted bya user presenting a single finger to the input device, to identifytarget strokes substantially having a shape consisting of a firstsegment inputted by only the single finger before the single fingerinputs a second segment, the first segment connected at an angle withthe second segment, the first segment having a different direction thanthe second segment; each time a target stroke is identified by theanalyzing, determining a count of fingers newly presented via the inputdevice substantially at a time when the target stroke's second segmentis completed by the user via the input device, wherein the one or morenewly presented fingers are associated with the target stroke; andwhenever a target stroke ends, performing a first step or a second stepaccording to the corresponding determined count of fingers for theending target stroke, wherein at least some of the target strokes have acount of a first number, and at least some of the target strokes have acount of second number, and wherein when the count of a target stroke isthe first number the first step is performed, and when the count is thesecond number the second step is performed and the first step is notperformed.
 15. A method according to claim 14, wherein when the count ofthe ending target stroke is two a third step is performed.
 16. A methodaccording to claim 15, wherein when the count of the ending targetstroke is three a fourth step is performed.
 17. A method according toclaim 15, wherein the angle is substantially ninety degrees.
 18. Amethod according to claim 15, wherein the input device comprises atouch-sensitive surface, and the discrete unbroken strokes are inputtedby corresponding discrete periods of a single finger initiating newcontact with the touch-sensitive surface and remaining in contact withthe touch-sensitive surface until the contact ends.
 19. A methodaccording to claim 15, wherein the steps correspond to features of amenu.
 20. A method according to claim 19, wherein the features comprisewhich items of the menu to select or display.